Immune checkpoint therapy (ICT) has transformed cancer treatment in recent years; however, treatment response is not uniform across tumor types. The tumor immune microenvironment plays a critical role in determining response to ICT; therefore, understanding the differential immune infiltration between ICT-sensitive and ICT-resistant tumor types will help to develop effective treatment strategies. We performed a comprehensive analysis of the immune tumor microenvironment of an ICT-sensitive tumor (melanoma, n = 44) and an ICT-resistant tumor (pancreatic cancer, n = 67). We found that a pancreatic tumor has minimal to moderate infiltration of CD3, CD4, and CD8 T cells; however, the immune infiltrates are predominantly present in the stromal area of the tumor and are excluded from tumoral area compared with melanoma, where the immune infiltrates are primarily present in the tumoral area. Metastatic pancreatic ductal adenocarcinomas (PDACs) had a lower infiltration of total T cells compared with resectable primary PDACs, suggesting that metastatic PDACs have poor immunogenicity. Further, a significantly higher number of CD68⁺ macrophages and VISTA⁺ cells (also known as V-domain immunoglobulin suppressor of T cell activation) were found in the pancreatic stromal area compared with melanoma. We identified VISTA as a potent inhibitory checkpoint that is predominantly expressed on CD68+ macrophages on PDACs. These data suggest that VISTA may be a relevant immunotherapy target for effective treatment of patients with pancreatic cancer.