This phase II trials studies how well pembrolizumab and vactosertib work after standard of care chemotherapy in patients with colorectal cancer that has spread to the liver that can be removed by surgery (resectable hepatic metastases). Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Vactosertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving pembrolizumab and vactosertib after standard of care chemotherapy, but before liver metastases surgery, may help shrink the cancer prior to surgery. This study also investigates pembrolizumab and vactosertib after liver metastases surgery, decrease the risk of the cancer recurring (coming back).