BigQuery Service Module for VOIP Data Management and Reporting

The project automate the generation of monthly reports related to VOIP data, upload them to BigQuery for analysis, and manage data efficiently. It combines API data, database queries, and cloud services to streamline reporting processes.

Table of contents

Data Retrieval and Processing:

This Node.js based application fetches VOIP-related data from external APIs and processes them into structured formats (e.g., CSV files).

async function customRequest(requestURL){

    const httpsAgent = new https.Agent({ rejectUnauthorized: false }); // ssl sorunu icin gerekiyor
    try {
        const response = await axios.get(requestURL, { httpsAgent });
        return response.data;
    } catch (error) {
        if (error.response) {
            console.log(error.response.data);
            console.log(error.response.status);
            console.log(error.response.headers);
        } else if (error.request) {
            console.log(error.request);
        } else {
            console.log('Error', error.message);
        }
        console.log(error);
    }

}astro.config.ts

BigQuery Integration:

In this step fetched and structured data integrates with Google Cloud’s BigQuery service to upload and manage datasets and tables.

class bq{
    constructor(){
        this.bigQuery = new BigQuery({
            projectId: '...',
            keyFilename: __dirname + '/.../'
        });


    }
    //WRITE_APPEND  WRITE_TRUNCATE
    async upload(datasetName,tableName,filename,type){
        if(type === "X1"){
            const metadata = {
                sourceFormat: 'CSV',
                skipLeadingRows: 1,
                autodetect: false,
                allowJaggedRows : true,
                writeDisposition:'WRITE_APPEND',
                schema: {
                    fields: [
                        {name: 'x1', type: 'STRING'},
                        {name: 'x2', type: 'FLOAT64'},
                        {name: 'x3', type: 'FLOAT64'},
                        {name: 'x4', type: 'FLOAT64'},
                        {name: 'x5', type: 'FLOAT64'},
                        {name: 'x6', type: 'FLOAT64'},
                        {name: 'x7', type: 'FLOAT64'},
                        {name: 'x8', type: 'DATE'},
                        {name: 'x9', type: 'DATE'},
                    ],
                },
            };
            await this.bigQuery.dataset(datasetName).table(tableName).load(filename,metadata);
        }
      }astro.config.ts

CSV Report Generation:

The processed data is written into CSV files and store in google bucket file.

      const newAllArray = [];
      const csvWriter = createCsvWriter({
        path: `./uploads/${fileName}${startOfMonth}.csv`,
        header: [
        {id: 'providername', title: 'provider'},
        {id: 'customerName', title: 'customer'},
        {id: 'Expense', title: 'Expense'},
        {id: 'In_Hr_rate', title: 'In_Hr_rate'},
        {id: 'Out_Hr_Profit', title: 'Out_Hr_Profit'},
        {id: 'startOfMonth', title: 'startOfMonth'},
        {id: 'endOfMonth', title: 'endOfMonth'},
                ]
            });
      await csvWriter
          .writeRecords(newAllArray)
          .then(()=> console.log('The CSV file was written successfully'));
      return `${fileName}${startOfMonth}.csv`astro.config.ts

Scheduled Tasks:

A cron job runs on the 1st of every month to fetch VOIP data for the previous month, process it, and generate CSV reports.

 new CronJob('40 11 1 * *', async function() {
     console.log("RUNNING CRON MONTHLY REPORT");
     const datex = moment().subtract(1, 'months').format('YYYY-MM-DD');
     var startOfMonth = moment(datex).startOf('month').format('YYYY-MM-DD');
     var endOfMonth = moment(datex).endOf('month').format('YYYY-MM-DD');
     await getVoiceData5G(startOfMonth,endOfMonth);
     await deleteCSV();
     console.log("***FINISH***")
 }, null, true,'Europe');
 console.log("VERSION 1.1.0");astro.config.ts