1. [numerical analysis] Said of an algorithm or computational method that tends to blow up because of accumulated roundoff error or poor convergence properties.
2. Software that bypasses the defined operating system interfaces to do things (like screen, keyboard, and disk I/O) itself, often in a way that depends on the hardware of the machine it is running on or which is nonportable or incompatible with other pieces of software.
In the IBM PC/mess-dos world, there is a folk theorem (nearly true) to the effect that (owing to gross inadequacies and performance penalties in the OS interface) all interesting applications are ill-behaved.
See also bare metal. Opposite: well-behaved, compare PC-ism.