I remember hearing about how Microsoft has rooms full of automated tests of running other people's programs on Windows. I've also heard about browsers using web sites as tests. I guess beta testing in general is about this kind of thing, just involving people in the loop.
Seems like to test your own library, if licensing allows it, you could actually include 3rd-party applications using the library as part of your own automated test suite, including their own unit tests. Might not be so bad in some cases.