1 Introduction
Use Zip to speed up CocosWeb loading
Some time ago, I used Cocos3.8 to do a cloud showroom project that required publish to the Web platform (WeChat H5 & browsers).
This project uses gltf models, the gltf models are split into many meshes and materials.
In Cocos, gltf is parsed and split to Cocos assets. After publishing to the Web, loading such as this gltf will use hundreds of network requests. Despite having fast internet speed, the loading is still slow because there is to many network requests.
2 Causes and Solutions
Why are there so many requests
- Create a new project, take a gltf from a ThreeJS demo, place it directly in the
resourcesfolder for easy preloading in the demo. This gltf contains a total of 28 materials, 32 meshes, and some bones & textures.
- Create a Start scene for preloading resources, and a Game scene for displaying the model.
- Create a Start script to perform a simple preloading of resources, when loaded then switch to the Game scene.
import {_decorator, Component, director, Label, settings, ProgressBar, resources, assetManager, Settings} from 'cc';
const {ccclass, property} = _decorator;
@ccclass('Start')
export class Start extends Component {
@property(ProgressBar)
progressBar: ProgressBar;
@property(Label)
barLab: Label = null;
async start() {
// Load the resources root directory
await this.preload([
{
path: "/",
type: "dir",
},
]);
director.loadScene("Game");
}
/**
* Preloaded resource
*/
preload = (pkg) => {
return new Promise<void>((resolve, reject) => {
const pathArr = [];
pkg.forEach((asset) => {
if (typeof asset == "string") {
return pathArr.push(asset);
}
switch (asset.type) {
case "dir":
resources.getDirWithPath(asset.path).forEach((v) => pathArr.push(v.path));
break;
default:
pathArr.push(asset.path);
}
});
resources.load(
pathArr,
(finish: number, total: number, _) => {
const pro = finish / total;
if (pro < this.progressBar.progress) {
return;
}
this.progressBar.progress = pro;
this.barLab.string = `Loading ${(pro * 100).toFixed(0)}%`;
},
async (error) => {
if (error) console.error(error);
resolve();
}
);
});
}
}
- Launch game and checking the Network Panel in the localhost, the number of network requests is 379, with 216 related to the gltf. Packaged and published to the Web with all JSON files merged, load the gltf only required 35 network requests.
localhost
Packaged & merged JSON
-
We have only one gltf, but it required 35 network requests to load.
-
The reason is that Cocos converts gltf resources to Cocos assets, disintegrates Mesh, materials, etc., and each resource has a Json file that records attribute dependencies in addition to the resource itself
How to solve
- Package the entire bundle, such as packaging it into a zip file. In the game, load the required bundle’s zip file for decompression. Subsequently, retrieve resources directly from the decompressed file.
3 Zip and JsZip
Zip
No need for further explanation since everyone is likely familiar with it.
Using JsZip
Refer directly to the npm platform documentation for JsZip.
jszip
https://www.npmjs.com/package/jszip
The documentation may seem abstract, so it’s better to follow the steps provided below for practical experience.
4 Exploring the Mystery of Cocos Resource Loading
- By examining the Network, it can be observed that Cocos downloads resources through a
download-file. tsfile, which can be traced by moving the mouse overdownload-file. tsto view its function call stack. The main components involved aredownload-file. tsanddownloader. ts, which are part of the resource download pipeline. Then opening the source code, we can delve into this process.
- As you can see in the code, most files are downloaded using the
downloadFilefunction, which is the function indownload-file.ts, which usesXMLHttpRequestto download the file
- The current practice in indicates that the majority of resource downloads rely on
XMLHttpRequestin Cocos, To optimize this process and minimize time consumption, we can intercept the requests and redirect them to the corresponding zip package.
5 Loading Your Zip Package
Loading Your Zip
- Create a
ZipLoader.tsand use it as a singleton. - The zip is loaded directly using the Cocos built-in API
- Use
Promisein conjunction with externalasync/awaitto simplify the control flow.
import {assetManager} from "cc";
import JSZIP from "jszip";
export default class ZipLoader {
static _ins: ZipLoader;
static get ins() {
if (!this._ins) {
this._ins = new ZipLoader();
}
return this._ins;
}
/**
* Download a single zip file as buffer
* Why is there a '.zip' suffix here that we'll talk about later, it's for automation
* @param path filePath
* @returns buffer
*/
downloadZip(path: string) {
return new Promise((resolve) => {
assetManager.downloader.downloadFile(
path + '.zip',
{xhrResponseType: "arraybuffer"},
null,
(err, data) => {
resolve(data);
}
);
});
}
/**
* load and unzip file
* @param path filePath
*/
async loadZip(path: string) {
const jsZip = JSZIP();
// donwload
const zipBuffer = await this.downloadZip(path);
// unzip
const zipFile = await jsZip.loadAsync(zipBuffer);
}
}
- Add code to
Start.tsbefore - Note the following points
- I use automatic compression upload plug-in, will auto modify the server field, server is the project published root directory with protocol and domain name, such as
https://xxx.com/cc_project/version/ - The plug-in will inject the package that requires zip loading into the window
- Such as
window["zipBundle"] = ["internal", "main", "resources"]; - All the bundles here are remote so just load the files in remote and the zip file is in the same directory as the bundle folder
/* ... */
@ccclass('Start')
export class Start extends Component {
/* ... */
async start() {
// automatic compression upload plug-in, will auto modify the server field
// and inject the package that requires zip loading into the window
// Such as
// window["zipBundle"] = ["internal", "main", "resources"];
const remoteUrl = settings.querySettings(Settings.Category.ASSETS, "server");
const zipBundle = window["zipBundle"] || [];
// All the bundles here are remote so just load the files in remote and the zip file is in the same directory as the bundle folder
// The zip file and bundle folder are in the same directory
const loadZipPs = zipBundle.map((name: string) => {
return ZipLoader.ins.loadZip(`${remoteUrl}remote/${name}`);
});
// wait zip loaded
await Promise.all(loadZipPs);
// load resources dir
await this.preload([
{
path: "/",
type: "dir",
},
]);
director.loadScene("Game");
}
/* ... */
}
Does not customize the engine, intercepting Cocos download
After referring to the Cocos documents, there is no efficient method available to achieve this requirement in batches, and because the Cocos engine is updated frequently, I personally like to use new engines and new functions, so a personal decision not to customize the engine, and directly adopt the method of intercepting Cocos loading to replace the loaded resources with my own zip package.
The examination of the source code apart from the image resource, other resources are loaded through the XMLHttpRequest, Therefore, it is a straightforward process, we directly intercept the XMLHttpRequest on the line directly.
So you ask me how to intercepting a browser Native object, this is JavaScript, JavaScript can do anything!
Intercepting ‘open’ and 'send’
- Without further ado, one can intercept an
XMLHttpRequestusing the following method to perform certain operations:
// Intercepting 'open'
const oldOpen = XMLHttpRequest.prototype.open;
// @ts-ignore
XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
return oldOpen.apply(this, arguments);
}
// Intercepting 'send'
const oldSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = async function (data) {
return oldSend.apply(this, arguments);
}
- Adding code for parsing zip files, extracting code from the zip to complete paths:
/* ... */;
const ZipCache = new Map<string, any>();
export default class ZipLoader {
/* ... */
constructor() {
this.init();
}
/* ... */
/**
* Loading and parsing Zip files
* @param path filaPath
*/
async loadZip(path: string) {
const jsZip = JSZIP();
const zipBuffer = await this.downloadZip(path);
const zipFile = await jsZip.loadAsync(zipBuffer);
// Parsing the zip file, concatenating paths, bundle names, and file names, and storing them directly in a Map
zipFile.forEach((v, t) => {
if (t.dir) return;
ZipCache.set(path + "/" + v, t);
});
}
init() {
// Intercepting 'open'
const oldOpen = XMLHttpRequest.prototype.open;
// @ts-ignore
XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
return oldOpen.apply(this, arguments);
}
// Intercepting 'send'
const oldSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = async function (data) {
return oldSend.apply(this, arguments);
}
}
- Within the intercepted
openandsendfunctions, cancel network requests and redirect them to cached zip resources. - Since the response property of XMLHttpRequest is read-only, we utilize
Object.getOwnPropertyDescriptorandObject.defineProperty. - Let’s dive straight into the code:
/* ... */
const ZipCache = new Map<string, any>();
const ResCache = new Map<string, any>();
export default class ZipLoader {
/* ... */
constructor() {
this.init();
}
/* ... */
init() {
const accessor = Object.getOwnPropertyDescriptor(XMLHttpRequest.prototype, 'response');
Object.defineProperty(XMLHttpRequest.prototype, 'response', {
get: function () {
if (this.zipCacheUrl) {
const res = ResCache.get(this.zipCacheUrl);
return this.responseType === "json"
? JSON.parse(res)
: res;
}
return accessor.get.call(this);
},
set: function (str) {
// console.log('set responseText: %s', str);
// return accessor.set.call(this, str);
},
configurable: true
});
// Intercepting 'open'
const oldOpen = XMLHttpRequest.prototype.open;
// @ts-ignore
XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
// Record resource if it exists
if (ZipCache.has(url as string)) {
this.zipCacheUrl = url;
}
return oldOpen.apply(this, arguments);
}
// Intercepting 'send'
const oldSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = async function (data) {
if (this.zipCacheUrl) {
// Skip parsing if cached
if (!ResCache.has(this.zipCacheUrl)) {
const cache = ZipCache.get(this.zipCacheUrl);
if (this.responseType === "json") {
const text = await cache.async("text");
ResCache.set(this.zipCacheUrl, text);
} else {
// Parsing using cocos set responseType directly for zip
const res = await cache.async(this.responseType);
ResCache.set(this.zipCacheUrl, res);
}
}
// Call onload after parsing and avoid making real network requests
this.onload();
return;
}
return oldSend.apply(this, arguments);
}
}
}
- Bundling the project and manually compressing the bundle folder for testing reveals three downloaded zip resources, with numerous JSON and binary folders not downloaded.
- Only two texture images were obtained for testing GLTF-related files, allowing successful entry into the Game scene.
- This demonstrates the effectiveness of the earlier code modifications, reducing the number of requests for loading the GLTF file from 35 times to 3 times.
6 Automated Publishing
- Developing a Cocos plugin for bundling and automatically compressing bundle files into zip formats is a straightforward endeavor.
- Simply create a building plugin and script file, as shown below:
import * as fs from "fs";
import JSZIP from "jszip";
//Read directories and files
function readDir(zip, nowPath) {
const files = fs.readdirSync(nowPath);
files.forEach(function (fileName, index) {
console.log(fileName, index);
const fillPath = nowPath + "/" + fileName;
const file = fs.statSync(fillPath);
if (file.isDirectory()) {
const dirlist = zip.folder(fileName);
readDir(dirlist, fillPath);
} else {
// Exclude image files, as explained below
if (fileName.endsWith(".png") || fileName.endsWith(".jpg")) {
return;
}
zip.file(fileName, fs.readFileSync(fillPath));//压缩目录添加文件
}
});
}
// Initiate file compression
export function zipDir(name, dir, dist) {
return new Promise<void>((resolve, reject) => {
const zip = new JSZIP();
readDir(zip, dir);
zip.generateAsync({
type: "nodebuffer",
compression: "DEFLATE",
compressionOptions: {
level: 9
}
}).then(function (content) {
fs.writeFileSync(`${dist}/${name}.zip`, content, "utf-8");
resolve();
});
});
}
- In the hooks
onAfterBuildsection - insert the compression script content to run post other operations and prior to resource upload
- particularly for web templates:
export const onAfterBuild: BuildHook.onAfterBuild = async function (options: ITaskOptions, result: IBuildResult) {
// Perform operation only on specific templates
if (options.platform !== "web-mobile") return;
// Modify scripts, obfuscate code, compress resources, etc.
/ ... /
if (fs.existsSync(result.dest + "/remote")) {
await Promise.all(
fs.readdirSync(result.dest + "/remote")
.map((dirName) => {
return zipDir(dirName, result.dest + "/remote/" + dirName, result.dest + "/remote");
})
)
}
/ ... /
// Upload
};
7 A Simple Optimization
Upon inspecting the source code and utilizing the Network tool,
it was noted that Cocos engine loads images by creating Image objects instead of using XMLHttpRequest.
have refrained from delving into replacing image loading with custom zipping methods for now,
as no such implementation has been carried out yet.
As a result, during zip packaging, PNG/JPG files are filtered out to reduce the size of the zip package,
containing only necessary files within the zip.