Personalized Vaccine with Checkpoint Inhibitor for Breast Cancer

Inclusion Criteria: * Written informed consent and Health Insurance Portability and Accountability Act (HIPAA) authorization for release of personal health information * Must be age \>= 18 years * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 within 14 days prior to tissue consent * Absolute neutrophil count \> 1500/mcL (obtained within 14 days prior to vaccine administration) * Absolute lymphocyte count \>= 600 cells/µl (obtained within 14 days prior to vaccine administration) * Platelets \> 100,000 mm (obtained within 14 days prior to vaccine administration) * Hemoglobin \> 9.0 g/dL (obtained within 14 days prior to vaccine administration) (NOTE: The use of transfusion or other intervention to achieve hemoglobin \[Hgb\] \> 9.0g/dl is acceptable) * Serum creatinine =\< 1.5 x upper limit of normal (ULN) or calculated creatinine clearance \>= 60 mL/min using Cockcroft-Gault equation for patients with creatinine levels \> 1.5 x institutional ULN (obtained within 14 days prior to vaccine administration) * Total bilirubin =\< 1.5 x ULN OR direct bilirubin =\< 1 x ULN (obtained within 14 days prior to vaccine administration) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x ULN unless liver metastases are present, in which case they must be =\< 5 x ULN (obtained within 14 days prior to vaccine administration) * Bilirubin =\< 1.5 X ULN (except in participants with documented Gilbert's disease, who must have a total bilirubin =\< 3.0 mg/dL) (obtained within 14 days prior to vaccine administration) * International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN unless patient is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants (obtained within 14 days prior to vaccine administration) * Activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless patient is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants (obtained within 14 days prior to vaccine administration) * TNBC as defined by estrogen receptor (ER)/progesterone receptor (PR) =\< 10% if Allred =\< 3; Her2/neu negative as defined by scores of 0 or 1+ by immunohistochemistry (IHC) or 2+ by IHC associated with a fluorescence in situ hybridization (FISH) ratio of \< 2.0 or \< 6 Her2 copies per cell * Patients with metastatic or inoperable locally advanced disease: Metastatic or inoperable locally advanced disease is defined as either histologically confirmed metastatic breast cancer by biopsy; or locally advanced breast cancer that, in the opinion of the treating physician, is not amenable to curative intent surgical resection; or, radiological or clinical evidence suggestive and supportive of metastatic disease * Documented metastatic biopsy is not required provided the patient has a prior diagnosis of TNBC that otherwise meets the eligibility criteria * Eligible patients must have =\< 3 lines of chemotherapy in the metastatic/advanced disease setting. For patients who have relapsed within 6 months of systemic therapy given within curative intent, that therapy will count as a line of metastatic therapy. Last cycle of cytotoxic therapy must be \>= 21 days prior to cycle (C)1 day (D)1 of vaccine. Last cycle of checkpoint inhibitor therapy be \>= 28 days prior to C1D1 of vaccine. Last dose of radiotherapy must be \>= 14 days prior to C1D1 of vaccine * Prior checkpoint inhibitor is permitted. Patients who are known to have PD-L1 positive with combined positive score (CPS) \>= 10 will be required to have had pembrolizumab therapy prior to enrollment * Patients who are metastatic or inoperable, locally advanced will only be eligible for the Phase 1b combination cohort. They will not be eligible for the Phase 1a dose escalation cohort. Patients who have received prior anti-CTLA-4 antibody will preferentially be enrolled to cohort B (combination of TMV vaccine with pembrolizumab). Patients who have received prior PD-L1 or PD-1 antibody therapy will preferentially be enrolled to cohort C (combination of TMV vaccine with ipilimumab). Patients with metastatic disease who have received no prior immune checkpoint inhibitor therapy will be assigned to a treatment arm based on available slots and discretion of treating physician * Patients with early stage TNBC: Early stage TNBC is defined as clinical or pathologic Stage I-III TNBC * After resection of disease in the breast and axilla, early stage patients are eligible for either the Phase 1a dose escalation of TMV vaccine monotherapy Cohort A or the Phase 1b combination arm of the vaccine with immune checkpoint inhibitor (ICI) * Patients will be required to have completed adjuvant radiotherapy (if indicated) \>= 14 days prior to initiation of vaccine on trial * Patients who have residual disease after completion neoadjuvant therapy that proceed with adjuvant capecitabine can enroll \>= 28 days after completion of final dose of capecitabine. Patients electing to enroll onto the Phase 1a Cohort A monotherapy arm can enroll prior to initiation of capecitabine, however must not initiate capecitabine prior to the Week 12 blood draw (measurement of immune biomarkers) on study * Patients who have a germline BRCA 1/2 mutation that meet the Food and Drug Administration (FDA) indication for use of adjuvant Olaparib can enroll \>= 28 days after completion of final dose of olaparib. Patients electing to enroll onto the Phase 1a Cohort A monotherapy arm can enroll prior to initiation of olaparib, however must not initiate olaparib prior to the Week 12 blood draw (measurement of immune biomarkers) on study * Patients who undergo upfront surgery: Patients may initiate injection of vaccine \>= 28 days after completion of final cycle of adjuvant chemotherapy * Patients who have early stage breast cancer that have residual disease after completing neoadjuvant chemotherapy with the KEYNOTE 522 regimen (pembrolizumab at a dose of 200 mg every 3 weeks plus weekly paclitaxel and carboplatin for 4 cycles followed by pembrolizumab-doxorubicin-cyclophosphamide or of pembrolizumab-epirubicin-cyclophosphamide every 3 weeks for 4 cycles): * Patient enrolling onto Phase 1a Cohort A will initiate injection of vaccine \>= 28 days after completion of final cycle of standard of care adjuvant pembrolizumab * Patient enrolling onto Phase 1b Cohort B or C will initiate injection of vaccine \>= 28 days surgical resection. Patients who have received pembrolizumab as part of the preoperative KEYNOTE 522 regimen will preferentially be enrolled to cohort B (combination of TMV vaccine with pembrolizumab) and will receive up to 9 cycles of adjuvant pembrolizumab every 3 weeks as per standard of care. Vaccine will be administered every 2 weeks for 3 doses prior to the first three ICI cycles * Patients who have early stage breast cancer that have that have residual disease after completing neoadjuvant chemotherapy with either dose-dense doxorubicin-cyclophosphamide followed by paclitaxel or docetaxel-cyclophosphamide: * Patient enrolling onto Phase 1a Cohort A will initiate injection of vaccine \>= 28 days surgical resection * Patient enrolling onto Phase 1b Cohort B or C will initiate injection of vaccine \>= 28 days surgical resection. Patients will be assigned to receive either pembrolizumab (Cohort B) every 3 weeks for 6 cycles or ipilimumab (Cohort C) every 3 weeks for 4 cycles in combination with TMV vaccine based on available slots in each arm and discretion of treating physician. Vaccine will be administered every 2 weeks for 3 doses; the TMV vaccine will be administered prior to the ICI cycle if ICI is due. * Weight of tumor tissue for production of vaccine must be at least 1 gram. In metastatic patients, preferentially, invasive tumor in breast or lymph node tissue will be retrieved by excisional biopsy to ensure sufficient yield. In metastatic patients who undergo initiation of first-line standard of care systemic therapy off study (i.e. pembrolizumab and chemotherapy in a PD-L1 positive patient), then ideally collection of tumor tissue will occur prior to treatment initiation of standard of care therapy to maximize cellularity. Metastatic patients who have undergone mastectomy during curative treatment initially or have no invasive disease in the breast, chest wall, or accessible regional lymph nodes, will be evaluated for image-guided core biopsies of liver metastasis or video-assisted thorascopic wedge resection of lung metastasis * In patients with early stage TNBC undergoing upfront surgery, the tumor tissue will be retrieved during lumpectomy/mastectomy. In early stage patients who are identified as high risk of having residual disease after neoadjuvant chemotherapy or undergo upfront resection, tissue will be retrieved during planned lumpectomy or mastectomy * Measurable disease is not required in metastatic patients but patients must have sufficient tumor to yield 1g on biopsy to enable production of personalized TMV vaccine product * Patients will undergo germline testing to assess for a BRCA1/BRCA2 deleterious mutation. Knowledge of germline status is not required to enroll on the study * Able and willing to complete the entire study according to the study schedule * Patients must give written informed consent. A copy of the signed informed consent form will be retained in the patient's chart Exclusion Criteria: * Weight of the tumor tissue is less 1 gram * Clinically significant comorbid conditions such as cardiovascular disease or significant peripheral vascular (e.g., uncontrolled hypertension, myocardial infarction, unstable angina) within 6 months of study entry, serious cardiac arrhythmia requiring medication, and uncontrolled infection * No second malignancy except prior breast cancer or except non-melanomatous skin cancer within the past 5 years * Ongoing or planned systemic anti-cancer therapy or radiation therapy. Last cycle of cytotoxic therapy must be \>= 21 days prior to C1D1 of vaccine. Last cycle of checkpoint inhibitor therapy be \>= 28 days prior to C1D1 of vaccine. Last dose of radiotherapy must be \>= 14 days prior to C1D1 of vaccine * Pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the study, starting with the pre-screening or screening visit through 120 days after the last dose of study treatment * Has a known history of active tuberculosis (Bacillus Tuberculosis) * History of allogeneic organ transplant * Known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Patients with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least 4 weeks prior to the first dose of study treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for management of brain metastases for at least 7 days prior to study treatment. This exception does not include carcinomatous meningitis which is excluded regardless of clinical stability * Diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of study treatment, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid * Active autoimmune disease that has required systemic treatment in the past 2 years (i.e., with use of disease modifying agents, corticosteroids, or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency) is not considered a form of systemic treatment * Known history of non-infectious pneumonitis that required steroids or any evidence of active pneumonitis * Failure to recover from grade 3 or 4 toxicity from previous treatment * For the combination cohort: prior grade 4 immune-related adverse events due to previous ICI. Patients who experienced grade 2 or 3 toxicity with prior ICI therapy may enroll if toxicity reverted to =\< grade 1