Magento 2 – Upstream Sent Too Big Header While Reading Response Header from Upstream

First of all, to comply with action controller interface \Magento\Framework\App\ActionInterface::execute(), your action must return an instance of \Magento\Framework\Controller\ResultInterface (\Magento\Framework\App\ResponseInterface is also supported, but is legacy and will be removed in future releases of M2, when all core usages are refactored).

So choose from available implementations of \Magento\Framework\Controller\ResultInterface. The most suitable for custom REST API (assuming it operates with JSON) seems to be \Magento\Framework\Controller\Result\Json. However, if you need something even more custom, consider \Magento\Framework\Controller\Result\Raw.

Working sample:

<?php
namespace VendorName\ModuleName\Controller;

/**
 * Demo of authorization error for custom REST API
 */
class RestAuthorizationDemo extends \Magento\Framework\App\Action\Action
{
    /** @var \Magento\Framework\Controller\Result\JsonFactory */
    protected $jsonResultFactory;

    public function __construct(
        \Magento\Framework\App\Action\Context $context,
        \Magento\Framework\Controller\Result\JsonFactory $jsonResultFactory
    ) {
        parent::__construct($context);
        $this->jsonResultFactory = $jsonResultFactory;
    }

    public function execute()
    {
        /** @var \Magento\Framework\Controller\Result\Json $result */
        $result = $this->jsonResultFactory->create();
        /** You may introduce your own constants for this custom REST API */
        $result->setHttpResponseCode(\Magento\Framework\Webapi\Exception::HTTP_FORBIDDEN);
        $result->setData(['error_message' => __('What are you doing here?')]);
        return $result;
    }
}

The code above will result in response with HTTP status code 403 and body {"error_message":"What are you doing here?"}

Magento makes 3 copies of your files, one for the thumbnail, one for small, and the Normal sized one. if you are using cache this means per 1 product you could have up 9 copies, per single image added per product.

I ran an image optimizer to try and reduce my files sizes, be cautious as mass jpg optimizers don't always reduce the files and can create colour table errors, as I experienced recently with some jpg's that where 5000x3000px, 10mb per file, it reduced them by 12% but the colours where all messed up.

Follow the Magento guidelines for image sizes dimensions, yes will take a long time to produce, for the 3 size types, but if you create for all 3, the thumbnail and the small should only be a few kb each where as you main image can be 500kb avg. and vastly reduce your taken hdd space.