A Study of Pembrolizumab in People With Ultra-Rare Sarcomas

Inclusion: * Patients must have pathologically confirmed diagnosis of one of the following: 1. Pleomorphic liposarcoma 2. PEComa (perivascular epithelial cell tumor) 3. Epithelioid sarcoma 4. CIC-rearranged sarcoma 5. SEF/LGFMS: Sclerosing epithelioid fibrosarcoma - low grade fibromyxoid sarcoma * Molecular characterization of the tumor, if available, will be recorded. If no such molecular data are available, note as such. * Patient should have recurrent or metastatic disease not judged to be curable with other means * Patients must have progressed (in the opinion of the treating investigator) following the most recent line of therapy or stop prior therapy due to toxicity or patient choice. The reason for this progression or other reason for changing therapy must be documented, employing tumor measurements when available. * Definition of Measurable Disease * Measurable disease as per RECIST 1.1. Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions for a minimum of 3 months after completion of such therapy * Patients with treated brain metastases are eligible if follow up brain imaging after CNS-directed therapy shows no evidence of progression at least 4 weeks after the completion of therapy as shown by follow up imaging before study screening * One to 3 prior lines of therapy are permitted (including neoadjuvant/adjuvant or metastatic/recurrent disease) * Participants who have AEs due to previous anticancer therapies must have recovered to ≤Grade 1 or baseline. Participants with endocrine-related AEs who are adequately treated with hormone replacement or participants who have ≤Grade 2 neuropathy are eligible. Patients with alopecia and other toxicities considered clinically nonsignificant and/or stable on supportive therapy, as determined by the investigator, are also permitted on study. * Administration of killed vaccines is allowed * Age ≥ 18 years of age. * ECOG Performance Status 0-1 (Karnofsky 70-100) * Required organ function (specimens to be collected within 14 days of the start of the study intervention) * Adequate hematologic function, defined as: * Absolute neutrophil count (ANC) ≥ 1,000 cells/mm\^3 * Platelets ≥ 100,000 cells/mm\^3 * Hemoglobin ≥ 9 g/dl * Adequate renal function defined, as: * Total bilirubin ≤ 1.5 x institutional upper limit of normal (ULN) (patients with known Gilbert disease who have bilirubin level ≤ 3 x ULN may be enrolled) * AST and ALT ≤3 x institutional ULN * Adequate cardiac function, defined as: class II or better New York Heart Association (NYHA) Functional Classification. * For patients with known HIV, HBV, and/or HCV infection \[HIV, HBV, and HCV testing do not need to be performed as part of the study; the below language provides guidelines for inclusivity of patients with known HIV, HBV, and/or HCV infection\]: * HIV-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months and CD4+ T cell count of at least 350 cell/mm3 are eligible for this trial. * HIV-infected patients with a history of either Kaposi sarcoma or Castleman disease are excluded from this study * HIV-infected patients must not have had an AIDS defining opportunistic infection within the past 12 months * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. Exclusion: * No prior diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 14 days prior to the first dose of study drug. * No other active malignancy, other than breast or prostate cancer stable for at least 6 months on hormonal therapy, or CLL Rai stage 0. Cancer in situ will not be considered an active malignancy. * No active autoimmune disease that has required systemic treatment in the past 2 years except replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid) * No prior (non-infectious) pneumonitis/interstitial lung disease that required steroids or has current pneumonitis/interstitial lung disease. * Must have adequately recovered from major surgery (at least 4 weeks), without ongoing surgical complications. Patients should have had any minor procedures (e.g. port placement, percutaneous nephrostomy) at least 2 weeks prior to the first day of treatment. * Has a history or current evidence of any condition, therapy, or laboratory abnormality or other circumstance that might confound the results of the study, interfere with the participant's ability to cooperate with the requirements of the study participation for the full duration of the study, such that it is not in the best interest of the participant to participate, in the opinion of the treating investigator. * No known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. * Not pregnant or breastfeeding or expecting to conceive children within the projected duration of the study, starting with the screening visit through 120 days after the last dose of trial treatment. * No prior allogenic tissue/solid organ transplant. * Patients with documented leptomeningeal disease are excluded from study, even if treated. * No prior systemic anti-cancer therapy including investigational agents within 4 weeks, 2 weeks for kinase inhibitors or intravenous chemotherapy, prior to starting therapy on study * No investigational agent(s) administration or use of investigational device within 4 weeks prior to study intervention administration. * No prior radiotherapy within 2 weeks of start of study intervention or radiation-related toxicities requiring corticosteroids. Note: Two weeks or fewer of palliative radiotherapy for non-CNS disease is permitted. The last radiotherapy treatment must have been performed at least 7 days before the first dose of study intervention. * No prior therapy with an anti-PD-1, anti-PD-L1, or anti PD-L2 agent or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (eg, CTLA-4, OX-40, CD137). * No live vaccine or live-attenuated vaccine within 30 days before the first dose of study intervention. * No known bleeding diathesis (e.g. hemophilia, DIC) * No active infection requiring systemic therapy * No history of allergic reaction to the study agent(s), compounds of similar chemical or biologic composition to the study agent (s) (or any of its excipients).