That's a pretty sizable topic and would be difficult to cover extensively in this forum.
I guess the first question is whether they (you) are subject to any particular standards or regulations? If so, that may drive a lot of what you should do.
Presuming not, here's a quick touch on what I would consider best practices throughout the life cycle.
Do they have a standardized approach; i.e., a well-defined software development life cycle? Is there evidence they are following it? A defined SDLC in itself won't result in better software but if they have one and have the discipline to follow it, it's a pretty good indication they can consistently deliver good results.
Do they have requirements established for the software? If so, how are they managed? Do they take changes through the life cycle or do they cut corners when developing new requirements? Will they be developing a release specific for your use or is this more "shrink-wrapped" software that is purchased as is? You mention it's going to be a part of a larger system so how is the integration going to be managed (this goes on your side as much as theirs!).
What do they do for design? Depending on the type of software, this may be less relevant. If they have anything, how do they maintain it?
When they develop the software, do they have an experienced development team or do they contract it out or use "guns for hire"? Having an experienced development team ensures consistency. You might look to see if software development is in their 'core' business. If not, you can still get good software but it may warrant looking deeper.
Do they have a defined software development environment? This would include all commercial software used, all Integrated Development Environments (IDEs), a Configuration Management (CM) system, and an issue tracking system (hopefully integrated with the CM system and/or the IDE)?
What do they do for testing the software? Can they show that all requirements are thoroughly verified? Do they use appropriate methods (unit, integration, white box, black box) when testing? (I realize that would be difficult to fully assess from your side but if they do things right, they can talk you through it and demonstrate the logic / rationale for what they're doing.)
How do they manage changes? Do they have formal change tracking in place? Do they prioritize changes (with your inputs) and have a solid update / roll-out approach?
How do they manage releases? What information do they provide with each release? You should expect to be able to see all open issues with the release as well as a summary of changes that are incorporated in the release. How do they coordinate releases with you or do they just throw them out and expect you to pick them up? This is more critical if you're in a regulated environment.
Everything mentioned above touches the Configuration Management (CM) process. If they have a good CM processes, they should be able to demonstrate all of this. Thus, I would start by asking them if they have a defined CM process and use that to drive the audit.
Like I said, it's a huge topic and the above is just brushing the surface. You'd be well advised to use some expert help when auditing.