Accumulation of toxic amyloid oligomers is a key feature in the pathogenesis of amyloid-related diseases. Formation of mature amyloid fibrils is one defense mechanism to neutralize toxic prefibrillar oligomers. This mechanism is notably influenced by apolipoprotein E variants. Cells that produce mature amyloid fibrils to serve physiological functions must exploit specific mechanisms to avoid potential accumulation of toxic species. Pigment cells have tuned their endosomes to maximize the formation of functional amyloid from the protein PMEL. Here, we show that ApoE is associated with intraluminal vesicles (ILV) within endosomes and remain associated with ILVs when they are secreted as exosomes. ApoE functions in the ESCRT-independent sorting mechanism of PMEL onto ILVs and regulates the endosomal formation of PMEL amyloid fibrils in vitro and in vivo. This process secures the physiological formation of amyloid fibrils by exploiting ILVs as amyloid nucleating platforms.