I worked for consulting firm A for few years in NY. Joined B (big trading firm) 4 months ago. I signed a non-compete agreement with A. Few employees of A work for B. Now, A says I should not work for B due to agreement

Agreement "I promise and agree that for a period of 2 years after termination of this agreement, I will not engage in competition with the COMPANY by working directly or indirectly for any clients or vendors of COMPANY"

Also

I Never worked for B through A.
'A' never mentioned and it's not explicit anywhere that B is his client
The team that am working at 'B' has no consultants from A and they never worked/tried to work on this project

So,

Is he right? how to handle him?
I need experience letter in future for GC processing. How to handle him in a better way?