Pomalidomide and Nivolumab in People With Virus-Associated Malignancies With or Without HIV

* INCLUSION CRITERIA: * Histologically or cytologically proven selected virus-associated tumors that are systemic, metastatic or locally advanced and not amenable to curative treatment options or are relapsed/refractory to first-line therapy as appropriate for each tumor type as outlined below. Also, participants with eligible solid tumors, who after evaluation by an expert in the area are deemed to be potentially curable after extensive surgery, but refuse such surgical procedure due to associated disfigurement and/or morbidity, may be eligible for the study with the necessary informed consent. Pathology confirmation by NCI Laboratory of Pathology is needed for eligibility. The following tumor types listed below are eligible, and require assessing of virus infection of the tumor cells with EBV EBER by in situ hybridization (ISH), KSHV LANA , p16, and Merkel cell polyomavirus large T antigen by immunohistochemistry (IHC) to document the respective viral infection (EBV, KSHV, HPV, MCPvY); or detection of serum HBV surface antigen, anti-HBV core antibody, elevated HBV DNA viral load, positive HVC antibody or elevated HCV RNA viral load. The tumor types studied in the phase 1 trial will be as below. For tumors where \>95% are known to be virus-associated, such as cervical cancer, confirmation of virus status is not required. --EBV-positive Hodgkin lymphoma meeting the following criteria: * Relapsed or refractory de novo classical Hodgkin lymphoma having failed standard first-line therapy; and * Unresponsive or progressive disease after treatment with brentuximab vedotin or may be brentuximab vedotin na(SqrRoot) ve but is ineligible or unable to receive brentuximab vedotin; and * Unresponsive or progressive disease after checkpoint inhibitor therapy; and * Unresponsive or progressive disease after or is ineligible for autologous stem cell transplant (auto-SCT) --EBV-positive aggressive non-Hodgkin lymphomas meeting the following criteria: * Relapsed/refractory disease after standard first-line chemotherapy; and * Relapsed disease after autologous stem cell transplant if indicated for histology (i.e diffuse large B-cell lymphoma relapsed more than one year after first line treatment) or autologous stem cell transplant is not feasible; and * Relapsed after CAR-T cell therapy for HIV-negative participants only if indicated for histology (i.e., diffuse large B-cell lymphoma) or CAR-T cell therapy is not feasible * EBV-positive nasopharyngeal cancer unresponsive or progressive disease on or after platinum-containing chemotherapy and/or radiotherapy * EBV-positive gastric cancer that is unresponsive or progressive disease on or after first-line chemotherapy * EBV-positive leiomyosarcomas that is unresponsive or progressive disease on or after 2 systemic regimens (CCRT/platinum-taxane) * Kaposi sarcoma impairing physical wellbeing (for example, tumor edema, pain, skin ulceration or breakdown, oral disease impairing function), no active KSHV-associated multicentric Castleman disease in past 12 months, and one or more of the following: * Inadequate tumor response after 6 or more cycles of liposomal doxorubicin or paclitaxel or other active cytotoxic agents (i.e. etoposide, bleomycin, anthracyclines, vincristine, vinblastine); or * Progressive disease while receiving liposomal doxorubicin or paclitaxel or other active cytotoxic agents (i.e. etoposide, bleomycin, anthracyclines, vincristine, vinblastine); or * Intolerant of liposomal doxorubicin and paclitaxel * Primary effusion lymphoma unresponsive or progressive disease on or after first-line combination chemotherapy * HPV-positive head and neck cancer that is unresponsive or progressive on or after first-line combination chemotherapy +/- radiotherapy * HPV-positive cervical cancer that is unresponsive or progressive on at least one systemic regimen for recurrent (does not include initial CCRT) or metastatic disease. Tumor HPV testing will not be a requirement for study eligibility for cervical cancer. * HPV-positive anal cancer that is unresponsive or progressive on or after first-line combination chemotherapy +/- radiotherapy * HPV-positive vaginal cancer that is unresponsive or progressive on or after first-line chemotherapy * HPV-positive penile cancer that is unresponsive or progressive on or after surgery and first-line chemotherapy * HPV-positive vulvar cancer that is unresponsive or progressive on or after first- line combination chemotherapy * MCPyV-positive Merkel cell carcinomas that is relapsed or refractory after prior checkpoint inhibitor therapy * HBV- or HCV-associated hepatocellular carcinoma that is not amenable to local therapy or liver transplant and has progressed on first-line therapy with sorafenib or levatinib or atezolizumab+bevacizumab * For solid tumors, participants must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>=20 mm (\>=2 cm) by chest x-ray or as \>=10 mm (\>=1 cm) with CT scan, MRI, or calipers by clinical exam. * For hematologic malignancies, participants must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (lymph nodes must measure \>=15 mm in the short axis and extranodal lesions must measure \>=10 mm in the short axis with CT scan. For primary effusion lymphoma, body cavity effusions may be followed as measurable disease by CT scan. * For KS, participants must have measurable disease, defined as at least five measurable cutaneous KS lesions with no previous local radiation, surgical or intralesional cytotoxic therapy that would prevent response assessment for that lesion. * Prior immunomodulatory therapy and checkpoint inhibitor therapy is allowed if previously tolerated without severe toxicities. Participants may not have received chemotherapy, radiotherapy, monoclonal antibody therapy, or targeted therapy within 2 weeks. * Age \>=18 years. Because no dosing or adverse event data are currently available on the use of pomalidomide in combination with nivolumab in participants \<18 years of age, children are excluded from this study. * ECOG performance status \<=2 (Karnofsky \>=60%). * Participants must have adequate organ and marrow function as defined below: * leukocytes no lower limit * absolute neutrophil count \>=1,000/mcL * platelets \>=75,000/mcL * total bilirubin \<= institutional upper limit of normal (ULN), except for participants with Gilbert disease or in whom the elevated bilirubinemia is due to ART (must be grade \<= 2) * AST(SGOT)/ALT(SGPT) \<=3x institutional ULN * glomerular filtration rate (GFR) \>=30 mL/min/1.73 m\^2 -Participants with any HIV status are eligible; for HIV-positive participants: * Must be on antiretroviral therapy (ART) \>4 weeks and with evidence of viral suppression defined as HIV viral load \< 400 copies/mL * Must have no major (e.g. AIDS-defining) opportunistic infections within the last 6 months except for the following which will be allowed: * Esophageal candidiasis treated within last 6 months or currently improving with antifungal treatment * Oral and/or genital HSV treated within last 6 months or currently improving with antiviral treatment * Mycobacterium avium infection in last 6 months or that has been treated for at least 1 month * For participants with evidence of chronic hepatitis B virus (HBV) infection, participants must be on suppressive therapy. * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For participants with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. * Able to take aspirin 81mg daily or a substitute thromboprophylaxis such as low molecular weight heparin at a prophylactic dose. * Participants with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. * Participants with new or progressive brain metastases (active brain metastases) or leptomeningeal disease are eligible if the treating physician determines that immediate CNS specific treatment is not required and is unlikely to be required during the first cycle of therapy. * Participants with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial, such as carcinoma in situ or low-grade prostate carcinoma. * Participants may not have cardiac or pulmonary abnormalities severe enough that they would pose a danger to receive treatment. To mitigate risks for cardiac AEs, screening EKG, echocardiogram, CPK, and troponin will be required and significant abnormalities as determined by the PI will need to be evaluated by Cardiology prior to enrollment. Participants with pulmonary symptoms will be required to undergo pulmonary function testing and Pulmonary evaluation at the discretion of the PI prior to enrollment. * Current or history of systemic autoimmune disease requiring systemic immunosuppressive therapy will not be allowed. Note: the following will not be exclusionary: 1) the presence of laboratory evidence of autoimmune disease (e.g. positive antinuclear antibody titer or lupus anticoagulant) without associated symptoms; 2) clinical evidence of vitiligo or other forms of depigmenting illness; 3) mild autoimmunity not impacting the function of major organs (e.g. controlled Hashimoto thyroiditis, limited psoriasis) * Persons of childbearing potential (PCBP), defined as a woman who: 1) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal for at least 24 consecutive months, i.e., has had menses at any time in the preceding 24 consecutive months) must have a negative serum or urine pregnancy test with a sensitivity of at least 25 mIU/mL within 10-14 days prior to and again within 24 hours of starting pomalidomide and must either commit to continued abstinence from receptive vaginal intercourse or begin TWO acceptable methods of birth control, one highly effective method and one additional effective method AT THE SAME TIME, at least 28 days before she starts taking pomalidomide. PCBP must also agree to ongoing pregnancy testing. Men must agree to use a latex condom during penetrative vaginal intercourse with a PCBP even if they have had a vasectomy. All participants must be counseled at a minimum of every 28 days about pregnancy precautions and risks of fetal exposure. * Ability to understand and the willingness to sign a written informed consent document. EXCLUSION CRITERIA: * Participants who have had anticancer treatment within the last 2 weeks, unless the cancer treatment is for a malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial, such as local treatment for carcinoma in situ or hormonal therapy for prostate or breast carcinoma. * Participants who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> Grade 1) with the following exceptions: * Elevated triglyceride attributed to ART and/or HIV (must be \<= Grade 2) * Laboratory or clinical abnormalities that are assessed as more likely to be from the underlying tumors, HIV disease, or other non-treatment causes will not be considered an exclusion criterion. * Alopecia, neuropathy and ototoxicity (i.e., AEs that are not expected to improve within the washout period) * Participants who are receiving any other investigational agents. * Participants will be excluded if they are on systemic steroid therapy that cannot be discontinued, with the exception of the use of prednisone or equivalent \<0.125mg/kg/day as replacement therapy. Inhaled or topical steroids are permitted. * History of allergic reactions attributed to pomalidomide and/or nivolumab or compounds of similar chemical or biologic composition to pomalidomide and/or nivolumab. * Participants who have received prior allogeneic stem cell or organ transplant. * Participants with severe uncontrolled intercurrent illness. * Cirrhosis with Child-Pugh score of B or C * Participants with psychiatric illness/social situations that would limit compliance with study requirements. * Pregnant and nursing persons are excluded from this study because pomalidomide is a thalidomide analog. Thalidomide is a known human teratogen that causes severe birth defects or embryo-fetal death. These potential risks may also apply to nivolumab based on its mechanism of action and data from animal studies. In animal reproduction studies, administration of nivolumab to cynomolgus monkeys from the onset of organogenesis through delivery resulted in increased abortion and premature infant death.