Nivolumab and Ipilimumab with Cabozantinib for Rare Tumors

Inclusion Criteria: * Metastatic disease defined as new or progressive lesions on cross-sectional imaging or bone scan. Patients must have at least: * One measurable site of disease as per Response Evaluation Criteria in Solid Tumors (RECIST) version (v) 1.1 * One bone lesion on bone scan (tec99 or sodium fluoride \[NaF\] PET/CT, CT or MRI) for the bone-only cohort. * Histologically confirmed diagnosis of one of the following metastatic cohorts: * Small cell/ neuroendocrine carcinoma of the bladder (Cohort A)- All urothelial carcinomas with any amount of neuroendocrine differentiation (including small cell differentiation) will be included. If the tumor is purely neuroendocrine, metastasis from another site of origin should be clinically excluded * Adenocarcinoma of the bladder, or urachal adenocarcinoma, or bladder/urethra clear cell adenocarcinoma (Cohort B) - must be pure (per World Health Organization \[WHO\] definition), (i.e. urothelial carcinoma with glandular differentiation is not considered a pure adenocarcinoma * Squamous cell carcinoma of the bladder (Cohort C) - must be pure (i.e. urothelial carcinoma with squamous differentiation is not considered a pure squamous cell carcinoma) * Plasmacytoid urothelial carcinoma (Cohort D) - Tumor should show predominantly \> or equal \~ 50% plasmacytoid histology (including all types of discohesive growth, such as tumors with signet-ring and/or rhabdoid features as well) * Any penile cancer (Cohort E) * Sarcomatoid renal cell carcinoma (Cohort F) - Tumor should be predominantly sarcomatoid \~ 50% (including rhabdoid differentiation) is also unclassified renal cell carcinomas (RCCs): all (assuming they are high grade with metastasis) malignant angiomyolipomas are allowed * Other miscellaneous histologic variants of the urothelial carcinoma, such as, but not limited to (Cohort G) : Micropapillary (Tumor should show predominantly \> or equal 50% micropapillary architecture), giant cell, lipid-rich, clear cell and nested variants (Tumor should predominantly \> or equal 50% show these features), large cell neuroendocrine carcinoma, lymphoepithelioma-like carcinoma and mixed patterns will be considered, as well as small cell neuroendocrine prostate cancer (Only treatment-naïve primary small cell of prostate with any amount of small cell component allowed. Post-treatment small cell prostatic carcinomas are not allowed), Malignant testicular Sertoli or Leydig cell tumors, and papillary and chromophobe RCC * Note: Translocation positive renal cell carcinoma patients are eligible. However, AREN1721 should be considered before this trial * Sarcomatoid urothelial carcinoma (Cohort H) - Tumor should show predominantly \~ 50% sarcomatoid differentiation * Renal medullary carcinoma (Cohort I) - Per World Health Organization (WHO) definition, ideally confirmed with immunostains * Bone-only metastatic GU tumors (non-prostate) (Cohort J) - All genitourinary histologies, except prostate are eligible * Renal Collecting Duct Carcinoma (Cohort K) - Per WHO definition (medullary involvement, predominant tubular morphology, desmoplastic stromal reaction, high grade cytology, infiltrative growth pattern, and absence of other renal cell carcinoma subtype or urothelial carcinoma) * Urethra carcinoma (Cohort L) - May be of any histology but if urothelial carcinoma then must be isolated to the urethra and not have metachronous or synchronous urothelial carcinoma of the bladder * H\&E slides from diagnostic tumor tissue for retrospective central pathology review * Patients may have received up to 2 systemic anti-cancer treatments or be treatment naive. Patients with small cell carcinoma should have received a platinum-based combination regimen either as neoadjuvant, adjuvant or first-line treatment). Patients in the bone-only cohort may be urothelial carcinoma histology but must receive standard cisplatin-based chemotherapy (if cisplatin-eligible) * Age \>= 18 years * Patients must be able to swallow oral formulation of the tablets * Karnofsky performance status \>= 80% * Absolute neutrophil count (ANC) \>= 1,000/mcL * Platelet count \>= 75,000/mcL * Total bilirubin =\< 1.5 x upper limit of normal (ULN). For subjects with known Gilbert's disease or similar syndrome with slow conjugation of bilirubin, total bilirubin =\< 3.0 mg/dL * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3.0 x institutional upper limit of normal (ULN) (or =\< 5 x ULN for patients with liver metastases or Gilbert's disease) * Creatinine =\< 1.5 x upper limit of normal (ULN) OR creatinine clearance \>= 40 mL/min/1.73 m\^2 (calculated using the Chronic Kidney Disease Epidemiology \[CKD-EPI\] equation or Cockcroft-Gault formula) for patients with creatinine levels above institutional normal * Hemoglobin \>= 9 g/dL (transfusion of packed red blood cells \[PRBCs\] allowed) * Serum albumin \>= 3.2 g/dL * Lipase and amylase =\< 2.0 x ULN and no radiologic (on baseline anatomical imaging) or clinical evidence of pancreatitis * Prior treatment with MET or VEGFR inhibitors is allowed. However, prior cabozantinib will not be allowed. Also, patients that have received both prior MET or VEGF and prior PD-1/PD-L1/CTLA-4 (sequentially or in combination) are also not allowed * No prior treatment with any therapy on the PD-1/PD-L1 axis or anti- CTLA-4/CTLA-4 inhibitors with the exception of patients with "urothelial carcinoma" histology (cohorts D, H, J, L) * Human immunodeficiency virus (HIV)-positive patients are eligible if on stable dose of highly active antiretroviral therapy (HAART), no clinically significant drug-drug interactions are anticipated with the current HAART regimen, CD4 counts are greater than 350 and viral load is undetectable * Patients with rheumatoid arthritis and other rheumatologic arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication only and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies etc. are eligible but should be considered for rheumatologic evaluation for the presence of target organ involvement and potential need for systemic treatment * Patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones or medications (e.g. thyroiditis managed with propylthiouracil \[PTU\] or methimazole) including physiologic oral corticosteroids are eligible * Patients who have evidence of active or acute diverticulitis, intra-abdominal abscess, and gastrointestinal (GI) obstruction, within 12 months are not eligible * Women of childbearing potential must have a negative pregnancy test =\< 7 days prior to registration * Women of childbearing potential include women who have experienced menarche and who have not undergone successful surgical sterilization (hysterectomy, bilateral tubal ligation, or bilateral oophorectomy) or are not postmenopausal. Post menopause is defined as amenorrhea \>= 12 consecutive months. Note: women who have been amenorrheic for 12 or more months are still considered to be of childbearing potential if the amenorrhea is possibly due to prior chemotherapy, antiestrogens, ovarian suppression or any other reversible reason * Pregnant women may not participate in this study because with cabozantinib, nivolumab, and ipilimumab have potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with cabozantinib, nivolumab, and ipilimumab, breastfeeding should be discontinued if the mother is treated with these agents * The patient has received no cytotoxic chemotherapy (including investigational cytotoxic chemotherapy) or biologic agents (e.g., cytokines or antibodies) within 2 weeks before the first dose of study treatment * The patient has received no radiation therapy: * To the lungs and mediastinum or abdomen within 4 weeks before the first dose of study treatment, or has ongoing complications, or is healing from prior radiation therapy * To brain metastasis within 3 weeks for whole-brain radiotherapy (WBXRT), and 2 weeks for stereotactic body radiation therapy (SBRT) before the first dose of study treatment * To the abdomen within 4 weeks before the first dose of study treatment, or has ongoing complications, or is healing from prior radiation therapy * To any other site(s) within 2 weeks before the first dose of study treatment * The patient has received no radionuclide treatment within 6 weeks of the first dose of study treatment * The patient has received no prior treatment with a small molecule kinase inhibitor within 14 days or five half-lives of the compound or active metabolites, whichever is longer, before the first dose of study treatment * The patient has received no prior treatment with hormonal therapy within 14 days or five half-lives of the compound or active metabolites, whichever is longer, before the first dose of study treatment. Subjects receiving gonadotropin-releasing hormone (GnRH) agonists and antagonists are allowed to participate * The patient has not received any other type of investigational agent within 14 days before the first dose of study treatment * The patient must have recovered to baseline or Common Terminology Criteria for Adverse Events (CTCAE) =\< grade 1 from toxicity due to all prior therapies except alopecia, neuropathy and other non-clinically significant adverse events (AEs) defined as lab elevation with no associated symptoms or sequelae * The patient may not have active brain metastases or epidural disease. Patients with brain metastases previously treated with whole brain radiation or radiosurgery who are asymptomatic and do not require steroid treatment for at least 2 weeks before starting study treatment are eligible. Neurosurgical resection of brain metastases or brain biopsy is permitted if completed at least 3 months before starting study treatment. Baseline brain imaging with contrast-enhanced CT or MRI scans for subjects with known brain metastases is required to confirm eligibility * No concomitant treatment with warfarin. Aspirin (up to 325 mg/day), thrombin or factor Xa inhibitors, low-dose warfarin (=\< 1 mg/day), prophylactic and therapeutic low molecular weight heparin (LMWH) are permitted * No chronic concomitant treatment with strong CYP3A4 inducers (e.g., dexamethasone, phenytoin, carbamazepine, rifampin, rifabutin, rifapentine, phenobarbital, and St. John's wort) or strong CYP3A4 inhibitors * Because the lists of these agents are constantly changing, it is important to regularly consult medical reference texts such as the Physicians' Desk Reference may also provide this information. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * The patient has not experienced any of the following: * Clinically-significant gastrointestinal bleeding within 6 months before the first dose of study treatment * Hemoptysis of \>= 0.5 teaspoon (2.5 mL) of red blood per day within 1 months before the first dose of study treatment * Any other signs indicative of pulmonary hemorrhage within 3 months before the first dose of study treatment * The patient has no tumor invading any major blood vessels * The patient has no evidence of tumor invading the GI tract (esophagus, stomach, small or large bowel, rectum or anus), or any evidence of endotracheal or endobronchial tumor within 28 days before the first dose of cabozantinib. Patients with rectal tumor masses are not eligible * The patient has no uncontrolled, significant intercurrent or recent illness including, but not limited to, the following conditions: * Cardiovascular disorders including: * Congestive heart failure (CHF): New York Heart Association (NYHA) class III (moderate) or class IV (severe) at the time of screening. * Concurrent uncontrolled hypertension defined as sustained blood pressure (BP) \> 150 mm Hg systolic, or \> 90 mm Hg diastolic despite optimal antihypertensive treatment within 7 days of the first dose of study treatment * The subject has a corrected QT interval calculated by the Fridericia formula (QTcF) \> 500 ms within 28 days before randomization. Note: if initial QTcF is found to be \> 500 ms, two additional electrocardiograms (EKGs) separated by at least 3 minutes should be performed. If the average of these three consecutive results for QTcF is =\< 500 ms, the subject meets eligibility in this regard * Any history of congenital long QT syndrome * Any of the following within 6 months before registration of study treatment: * Unstable angina pectoris * Clinically-significant cardiac arrhythmias (patients with atrial fibrillation are eligible) * Stroke (including transient ischemic attack \[TIA\], or other ischemic event) * Myocardial infarction * Cardiomyopathy * No significant gastrointestinal disorders particularly those associated with a high risk of perforation or fistula formation including: * Any of the following that have not resolved within 28 days before the first dose of study treatment: * Active peptic ulcer disease * Acute diverticulitis, cholecystitis, symptomatic cholangitis or appendicitis, or malabsorption syndrome * None of the following within 2 years before the first dose of study treatment: * Abdominal fistula or genitourinary fistula * Gastrointestinal perforation * Bowel obstruction or gastric outlet obstruction * Intra-abdominal abscess. Note: Complete resolution of an intra-abdominal abscess must be confirmed prior to initiating treatment with cabozantinib even if the abscess occurred more than 2 years before the first dose of study treatment * Disorders associated with a high risk of fistula formation including percutaneous endoscopic gastrostomy (PEG) tube placement are not eligible * No other clinically significant disorders such as: * Severe active infection requiring IV systemic treatment within 14 days before the first dose of study treatment * Serious non-healing wound/ulcer/bone fracture within 28 days before the first dose of study treatment * History of organ or allogeneic stem cell transplant * Concurrent uncompensated hypothyroidism or thyroid dysfunction within 7 days before the first dose of study treatment (for asymptomatic patients with an elevated thyroid-stimulating hormone \[TSH\], thyroid replacement may be initiated if clinically indicated without delaying the start of study treatment) * No history of major surgery as follows: * Major surgery within 3 months of the first dose of cabozantinib; however, if there were no wound healing complications, patients with rapidly growing aggressive cancers, may start as soon as 6 weeks if wound has completely healed post-surgery * Minor surgery within 1 month of the first dose of cabozantinib if there were no wound healing complications or within 3 months of the first dose of cabozantinib if there were wound complications excluding core biopsies and mediport placement * Complete wound healing from prior surgery must be confirmed before the first dose of cabozantinib irrespective of the time from surgery * No history of severe hypersensitivity reaction to any monoclonal antibody * No evidence of active malignancy, requiring systemic treatment within 2 years of registration * No history of allergic reactions attributed to compounds of similar chemical or biologic composition to cabozantinib, nivolumab, ipilimumab or other agents used in study * No positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (HCV antibody) indicating acute or chronic infection. If HBV sAG is positive, subsequent ribonucleic acid (RNA) polymerase chain reaction (PCR) must be negative * No patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids. These include, but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease